Andy Smith, 2019
People keep asking for my opinion over what is wrong with the rats... so I first wrote this in 2017. I have reviewed it now in 2019 and made no significant changes. If I am wrong on anything, please point it out.
I tested the APOPO rats in Mozambique back in 2003 – for the German demining INGO MgM which was genuinely interested in using them. I really wanted them to work and did everything possible to help but they could not perform and APOPO withdrew from the tests to avoid having to publish the results (publishing results no matter what they were had been a part of the contract, but they ignored that).
That was a long time ago, so maybe things have changed?
The nearest thing to a scientific assessment of current APOPO rat capability that I can find online is in the Journal of Applied Behavioural Analysis: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3120071/
This is not an independent report. It is an article in which the author has believed everything he was told and sought to explain things clearly to the readers, and there are some things he did not understand, so did not question.
Bear with me as I walk through the article and comment....
Rat training starts with “clicker and reward training” to elicit a Pavlovian response. The amount of explosive used is far greater than the traces being sought in the real world and mixed in water, so may have an unrealistically evaporative base releasing far greater amounts of vapour than a real target. The Click and the reward are immediately given. When they have learned to get food this way, the rats are moved on to 'tea eggs' (we are not told what form the target TNT in the 'eggs' took) and they have to perform well for two days before moving on.
So far, they have had to respond to a vapour without real-world distractions and success has been immediately rewarded with food. The use of food suggests that praise or pleasure from the handler is not enough – which may reflect a rat’s inability to relate to people (relative to dogs).
At this stage, a mechanistic cause and effect response has been established in an ideal environment. The Clicking is included because the rats will later associate it with deferred reward.
Field training then occurs in a dummy minefield with no vegetation at all. I immediately wonder how the vegetation is cut? I wonder because of the problem I had scalping the ground low enough for the rats to be dragged over in the early trials. They probably use a lawn mower over an area that is unrealistically smooth and flat. The 'tea-eggs' are buried – but it is not clear how long before the rats are brought or how careful to avoid contamination or 'human trails' this process is. A person standing/crouching and burying a target leaves a strong scent. Not all of the targets contained TNT, but if the rats could follow the handler’s scent to the possible positive places, they would not have to search, merely to confirm or dismiss an olfactory 'hotspot'. The absence of 'searching' is significant.
The ropes holding the rats between two handlers include one that is used to pull the animal when it pauses, gets distracted, false-alarms, or needs 'encouragement'. So the rats have not yet been taught how to move or diligently search, merely to react for reward.
They then move to a test area with real (safe) mines concealed. When it was last searched is not mentioned – so it could have been contaminated by scent traces of previous rats or people placing targets (they would need very extensive training areas if this were not true). At that stage, the rats are used to search two 3m x 10m wide boxes a day, so 60 sqm. The reward of food is delayed until they complete a 3m line but their success is confirmed with an audible 'Click' so that the rats know it is coming. The size of the box (length of the rope between rat handlers) is extended until they are searching 100 sqm boxes (10x10m) and only then do the handlers not know where the targets are. Information about how they are rewarded is absent at this point – so presumably the rats are rewarded for every indication. They have to find the targets (which may be none) in four 10x10m areas. We are not told how many targets there actually are – it could be 0 – 16, and they can false alarm 8 times (which is more than 50% because some boxes contain fewer than four targets). When the boxes were last used is not made clear, so the potential for them following previous scent traces has not been addressed. And the rats have still not been trained to search – they still need to be dragged back and forth across the area with a path a few cms wide that is beneath their noses, so possibly 'sniffed' if not exactly 'searched'.
Then they move to Mozambique for accreditation and there is the same problem of using a test area that has been used many times before. The scent of the person placing the targets and the scent of rats conducting previous searches all conspire to make the detail given not enough to convince anyone that the rats are doing more than following handlers and predecessors. The MAC in Mozambique had no money and no animal-search specialists so simply watched them in their own test area, where they probably did well. When the MAC asked about the tests conducted in 2003 by MgM and me, it seems that APOPO staff led them to believe that those tests had been successful, which was simply not true.
When the rats were moved to search real areas, the rewards system was changed so that rats were rewarded on a test mine buried each day (so definitely a contaminated target that, frankly, I can replicate with a cat by smearing cat-foot onto something and burying it in the garden – I tried it and it works!) and we are told that is 'sufficient'. In fact it’s a little like using a metal detector test-piece and believing that signaling on that means you will find the mines you are looking for (all real deminers know that it really does not). The area had all vegetation removed by a large armoured brush-cutter, then safe lanes for the handlers to stand in were manually searched and cleared. The box sizes were 10x20 sqm. The paths searched and cleared manually around and between will have been 2m wide – which is the International Standards requirement to allow safe access in the event of an accident. So to allow access, the paths around one 10x20 sqm box amount to 128 sqm (allowing for corners). The box is 200 sqm and the paths are 128 sqm. When boxes are adjacent, paths are shared so the proportion of manually searched and cleared areas to rat searched areas drops, but it is still high (and this is the reason why running search dogs in boxes is now rare – because dogs can be trained to move as required, there is no need to use manually prepared 'boxes'.) The manual search and clearance of the access paths, then marking them so that the position of rat indications can be 'kind-of' estimated would take rather a long time and cost a lot.
We are told that the rats searched 93,400 sqm in 2009 and found “41 mines”. If they found no explosive hazards other than mines, they probably missed them – because in my experience there are almost always other munitions in minefields. However, the rats were not used in minefields. I say this because they supposedly searched 2,278 sqm for every device found – and that is not a realistic reflection of mine density in any minefield. Then the article suggests that the entire area was searched manually afterwards to check they had not missed anything. Look at APOPO’s own estimate for how much land a man can search and clear. It is impossible to generalise with any accuracy, but let’s ignore them and say that a man can do 20 sqm a day (which is greater than the average they sometimes allow), so APOPO expended at least 4,670 deminer days after the rats. If a deminer works an average of 220 days a year (holidays, weekends, etc allowed for), they needed at least 42 full-time manual deminers for the six months of the year likely to have been available. This is in addition to other deminers needed to prepare the areas for further rat search and investigate any indications. Then you have to add team leaders, site managers, medics, drivers and logistics staff. Lets say an additional 75 in all, plus vehicles, accommodation and all ancillary costs. Did they have that kind of demining capacity?
Er, no, apparently nothing like.
So I cannot accept that they manually searched the whole area after using the rats.
APOPO rats are never trained to move in a search pattern and always need to be either dragged on a harness back and forth over flat and vegetation free ground between a pair of handlers - or tied to a stick and poked into suspect areas (and how the stick-holder knows if the rat is signaling when it is immersed in vegetation is anyone’s guess).
The data used in the article was all supplied by APOPO – and there was no independent oversight to show that they really did what they claim to have done. Internal inconsistency in their reported claims make it highly unlikely that they did what was claimed. All in all – the article in the Journal of Applied Behavioural Analysis only serves to raise credibility problems for APOPO.
Another supposedly 'rigorous' report on APOPO was published in 2016 by the Geneva International Centre for Demining (GICHD). http://www.gichd.org/fileadmin/GICHD-resources/rec-documents/APOPO-GICHD-Mine-Detection-Rats-30Jun2016.pdf.
This is a piece of shallow journalism that relies entirely on APOPO’s own data to 'prove' they are cost effective, then suggests that they “should improve their data collection system” (so throwing doubt on the very data it relies on). The publication is unattributed but it was apparently authored by someone who has friends amongst people involved with APOPO (although that may not be relevant, of course).
This article describes the deployment variant that uses a single rat tied onto a stick to search. This reminds one of some people’s blind faith in using a single dog – which NPA’s Mine Dog research centre in Bosnia proved wrong years back but which some never quite understood.
Their argument is, “There is no need to use a second dog or rat because if they are both trained the same way, they will both miss the same things”.
This is true – for a given value of 'true'. However, when conducting Quality Assurance on a search by searching again, you are not checking whether the detector/dog/rat can find the targets. That should have been done before any search started. The second detector check (with a metal-detector a dog or a rat) is simply checking whether the first search was conducted properly. A second dog searching the same area in a different pattern or a second search with a metal-detector provides assurance that the first detector was working as intended, or used as intended. Not bothering with Quality Assurance means that a faulty detector (or deployment method) will not be identified and that is simply irresponsible – AND is in direct conflict with all the principles of Quality Management built into the International Mine Action Standards (IMAS) and, ironically, pushed by GICHD.
The author of the GICHD article thinks that we were slow to adopt the use of dogs (MDD) almost twenty years ago. We did not resist the use of MDD – we resisted the use of unproven and ill-prepared dogs for the very good reason that they would not search reliably and so missed mines (I have many examples of this in the accident record). This is why NPA spent so much money working on finding out how best to train and deploy dogs at their Bosnia centre. Unfortunately, they found that the costs involved in using dogs responsibly in the field are often prohibitive. The GICHD author also failed to ask questions of those who had been involved in the only independent test of the rats ever conducted, so was not demonstrably thorough or professional about his background research and did not show due diligence.
The GICHD article supporting the rats was no better researched than the Behavioural Analysis article because it relied entirely on APOPO’s own incomplete data. The author carried out no independent testing (well designed or otherwise) and relied on in-house MAC 'testing'. This accreditation testing is often conducted by people without relevant expertise so is not valuable unless a report is published showing genuinely independent and scientifically rigorous testing. No such evaluation of the rats has been published. No genuinely disinterested and independent examination of the rate of rat success or the costs associated with their use could be referenced in the GICHD article because none has been done (or none has been made public anyway).
For GICHD to put their name to this endorsement is braver than they may realise.
The facts are simple.
1) Rats have short legs and can only be used in a smooth, obstruction free area.
2) Rats cannot be trained to move in a search pattern.
3) Rats have the capacity to detect explosive residue and can apparently be encouraged to do so with food rewards...
4) ...but there is no evidence that rats can respond to explosive residue in a hazardous area reliably.
5) There is no evidence that the method of deployment results in a thorough search of the area.
6) The speed of search sometimes claimed would make thorough search (tiny nose within 10 cms of target) physically impossible.
7) The cost of preparing the area where they are used, then dragging the rat back and forth over it has to be added to training and housing the rats.
8) While a rat may indicate on a hazard, it cannot expose and clear it. The cost of manually excavating and clearing any explosive hazard that the rat signals on must also be added to the total cost of clearance.
9) The ground has to be scalped for them to be dragged across it in a straight line, then safe-lanes for the handlers manually searched and cleared – so the total cost of using the rats includes the cost of an armoured vegetation cutter and a fully equipped demining team to prepare the area in which the rat will search.
10) There is absolutely no evidence that the cost is less than the cost of demining the area using the same assets but without rats.
11) Because the rat cannot be trained to search and indicate in a set way (which a dog can be reliably trained to do) there is no way for a Quality Assurance observer to know whether the rat is paying attention, so no way of knowing whether the rat has done anything of any value at all.
APOPO’s failure to permit/encourage/finance a genuinely independent performance assessment using tests that evaluate detection ability in several real situations suggests that they know the animals do not perform well enough outside their own 'loaded' test areas. A World Health Organisation assessment of the APOPO rats' ability to reliably sniff out human disease (a parallel use for the APOPO rats) found that they could not do so reliably, a fact which APOPO ignores in its publicity material.
Releasing land for children to play on that has not been searched thoroughly is immoral in any humanitarian activity – but seems to be considered a 'necessary evil' by some. These people are apparently unaware of the primary goal of HMA – which has always been to REDUCE the risk posed by explosive hazards to civilians. Encouraging them to use inadequately searched land that they had previously avoided definitely increases their risk.
I believe that APOPO is riding towards an inevitable fall. When one of the people they put at risk realises the potential for suing them in the Belgian/European courts, their failure to provide credible evidence for their performance claims may cost them dear. Litigation in the USA is also a possibility and with no-win no-fee specialists looking for work, it is certainly a growing risk. There are people waiting to give evidence for the prosecution – and enough published criticism to leave APOPO no defence of ignorance to hide behind (which would have been a tough call anyway). “All reasonable effort” and “tolerable risk” cannot be wriggled behind or used to divert blame to the National Mine Action Authority because the court where the case is heard will use its own definition of these terms. The fact is that even donors (who give credence to things they fund) should be getting nervous.
GICHD really should take down the 'report' on APOPO because they have given one man’s biased and unjustified opinion the credibility of anonymous publication under their banner, so it is the wealthy organisation that the lawyers will add to their attack. The rats don’t search reliably enough to give assurance that the area is searched. Anyone claiming otherwise has to prove that, or lose more than their karma when the claims start coming in.
Sadly, those of us with long memories know that the REST system was manoeuvred into the IMAS with no evidence of its effectiveness – and that REST has long been discredited by real-world failures. I think that REST probably could work in ideal circumstances – but I cannot remember ever having seen ideal circumstances in a mined area. 'Sometimes working' really is not good enough but being in the IMAS means that some naive newcomers will always be willing to throw more donor money down that drain again, so whoever wheedled that into the IMAS did us all a disfavour...
I am reliably informed that the same person who promoted REST is seeking to ensure that the use of a single dog (or rat) is formally added to the IMAS. It may not need saying that this man is paid by APOPO. His influence has already been effective with the GICHD adoption of the word 'animals' (replacing 'dogs') and the adoption of silly rat logos. He was behind the effort made to adjust the IMAS to permit the accreditation of single 'wonder animals' (dogs or rats) in early 2017. This was not permitted because of the absence of any evidence that it could work reliably. Giving IMAS support for the interests of APOPO would be as morally and professionally hazardous as its previous support for MECHEM’s MEDDS system (renamed REST by the rat advocate). Knowingly adding unproven and hazardous procedures to the IMAS puts Review Board members at risk of being held liable when trusting end-users believe they have done background research amounting to 'due diligence' – which, in APOPO’s case, there is apparently no evidence that anyone has.
The available records of demining accidents and 'missed-hazards' provide evidence that the use of a single dog to conduct a search is inadequate and has led to hazardous land being deemed 'safe'. The records are incomplete, so not comprehensive, but in the absence of other evidence these records can be expected to carry more weight during a legal dispute than the unsupported opinion of self-interested individuals.
APOPO is a nominally 'not for profit' organisation but I can find no public record of its accounts (apparently making this public is not necessary when based in Belgium). The APOPO publicity makes grand claims for 'cost-effectiveness' but a failure to publish their accounts makes that impossible for anyone to independently assess. It is possible that their real search and clearance work is all done manually and that the rats are just a lucrative publicity angle. If that is so, their rat cost-effectiveness claims are simply untrue. Whatever, I believe that its management and advisors are highly paid to ensure that its humanitarian purposes are met but, based on the publicity material I have seen, it is my opinion that they fail to achieve their humanitarian goal and so both mislead their donors and take money under an amoral equivalent of false pretences.
STATEMENT
“In the absence of any scientifically rigorous evidence that APOPO rats (and their deployment methods) are able to reliably search to a standard that ensures that everything reasonable has been done to locate explosive hazards to a defined depth in a hazardous area, it is my opinion that their use as a primary or sole search-tool in any hazardous area that will be used by civilians is unreasonable, does not meet the requirements of the International Mine Action Standards (IMAS), and puts the end-users of the land at unnecessary, so intolerable, risk.
The overbearing weight of available evidence is that reliance on the APOPO rat system for explosive hazard detection in areas for use by civilians puts the end-users of the land at unnecessary risk, so is both immoral and inhumane."
If I am wrong on any point, please correct me.